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Abstract 

Spacecraft analysts in the spacecraft control center for the COBE (Cosmic Background Explorer) 
satellite are currently utilizing a fault-isolation expert system developed to assist them isolate and 
correct faults in the communications link. This system, named CLEAR (Communications Link Expert 
Assistance Resource), monitors real-time spacecraft and ground system performance parameters in 
search of configuration discrepancies and communications link problems. If such a discrepancy or 
problem is isolated, CLEAR alerts the analyst and provides advice on how to resolve the problem swiftly 
and effectively. The CLEAR System is the first real-time expert system to be used in the operational 
environment of a satellite control center at the NASA Goddard Space Flight Center. 

CLEAR has not only demonstrated the utility and potential of an expert system in the demanding 
environment of a satellite control center, it has also revealed many of the pitfalls and deficiencies of the 
development of expert systems. One of the lessons learned from this and other initial expert system 
projects is that prototypes can often be developed quite rapidly, but operational expert systems require 
considerable effort. Development is generally a slow, tedious process that typically requires the special 
skills of trained programmers. 

Due to the success of CLEAR and several other systems in the control center domain, a large number of 
expert systems will certainly be developed to support control center operations during the early 1 990s. 
To facilitate the development of these systems, a project has been initiated to develop an integrated, 
domain-specific tool, named GenSAA (Generic Spacecraft Analyst Assistant), that will allow the 
spacecraft analysts to rapidly create simple expert systems themselves. By providing a highly graphical, 
point-arid- select method of system development , GenSAA allow the analyst to utilize and! or modify 
previously developed rule bases and system components, thus facilitating software reuse and reducing 
development time and effort. 


Introduction 

The Goddard Space Flight Center is responsible for 
managing the operations of numerous low-earth orbit 
satellites. These scientific satellites either have a 
dedicated control center (e.g. LANDSAT and the 
Hubble Space Telescope) or share computer 
resources in the Multi-Satellite Operations Control 
Center (e.g. Cosmic Background Explorer and Earth 
Radiation Budget Satellite). In either case, highly 
trained personnel, called flight operations analysts 
(FOAs), are responsible for the proper command, 
control, health and safety of the satellite. 

The satellite control centers operate round-the-clock 
throughout the lifetime of the spacecraft. There are 


typically multiple real-time communications events 
with each satellite daily. During these real-time 
communications events, the FOAs must: 

- establish and maintain the 
telecommunications link with the spacecraft, 

-monitor the spacecraft’s health and safety, 

- send commands or command loads to the 
satellite for on-board execution, 

- manage spacecraft resources (including on-board 
memory, batteries, and tape recorders), and 

- oversee the dumping of the scientific data from 
the on-board tape recorders to ground systems 
for processing and analysis. 
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To accomplish these activities, the analyst must 
combine a thorough understanding of the operation 
of the spacecraft and the ground systems with the 
current state of operations as indicated by numerous 
telemetry parameters displayedxm the consoles. 
During an event, the analyst typically monitors 
hundreds of telemetry parameter values on multiple 
display pages that may be updating several times per 
second. The monitoring of the operation of these 
satellites is a demanding, tedious task that requires 
well-trained individuals who are quick-thinking and 
composed under pressure. 

As spacecraft become more complex, the task of 
operating a satellite is becoming increasingly more 
difficult. The FOAs are reaching a level of 
saturation as more and mo re data must be monitored 
and analyzed during the real time supports. The 
need to automate some of these functions is 
apparent. 


The CLEAR System 

The Communications Link Expert Assistance 
Resource (CLEAR) is the first attempt at the 
Goddard Space Flight Center to utilize an expert 
system to automate a spacecraft analyst’s task. 
CLEAR is a fault-isolation expert system this is 
supporting real-time operations in the Payload 
Operations Control Center (POCC) for the Cosmic 
Background Explorer (COBE) mission. This system 
monitors the communications link between COBE 
and the Tracking and Data Relay Satellite (TDRS), 
alerts the analyst to any discrepancies or problems, 
and offers advice on how to correct them. It is the 
first expert system to become operational in a 
satellite control center at NASA/Goddard. 

CLEAR is a forward chaining, rule-based system 
that operates in the COBE Mission Operations 
Room (MOR). It monitors over 100 real-time 
performance parameters that represent the condition 
and operation of the spacecraft’s communications 
with die relay satellite. With this information, 
together with the knowledge of TDRS operations, 
COBE’s on-board communications system and the 
expected configuration of the scheduled event, 
CLEAR can accurately portray the status of the 
communications link. 

The CLEAR Expert System is currently supporting 
the COBE flight operations analysts for fault 
isolation. It is used routinely and is regarded as the 
fault-isolation “expert” for the COBEyTDRS 


telecommunications link. 

The user interface for CLEAR utilizes textual and 
graphical output in a tiled-window format to provide 
the user with information about the status of the 
COBE/TDRS/ground communications links. A 
graphics window displays all of the elements of the 
communications network from the COBE Spacecraft 
to the POCC with green lines representing healthy 
links between elements. If the performance 
parameters indicate that the communications link or 
processing system is degrading or down, then the 
associated icon will turn yellow or red, respectively. 
This display enables analyst s to assess the current 
status of the communications event in a quick 
glance. 

When CLEAR isolates a problem, a short 
description of the problem is displayed in the 
“problems” window. If multiple problems are 
found, the problem descriptions are ranked and 
displayed in descending order of criticality. CLEAR 
suggests actions for the analyst to take in order to 
correct the problem; however, the system does not 
take any corrective action itself. 

To further assist the analyst and to provide support 
for its advice, the CLEAR system provides an 
explanation facility. When the analyst selects a 
problem displayed in the problems window, CLEAR 
provides a detailed explanation of why the expert 
system believes that the problem exists. No 
backtracking or backward chaining is conducted 
since the system must continue to monitor the real 
time data and fire forward chaining rules. 

CLEAR maintains an event log to record histories 
and allow offline analysis of problems. The event 
log has proven to be quite useful for operational 
support of the mission and continued enhancement 
of the knowledge base. 

The CLEAR System operates on any of the seven 
PC/AT-type workstations that are used for console 
operations in the MOR. It is written in the C 
language and uses the C Language Integrated 
Production System (CLIPS) and a custom-developed 
graphics library. It currently has approximately 160 
rules. Additional rules may be added to monitor the 
tape-recorder dumps from the satellite to the 
Wallops ground station. 

CLEAR isolates approximately 70 different 
problems. The types of problems include: 
non-reception of data within the control center 
(system or communication problems, or data 
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reporting not activated); misconfigurations between 
the COBE MOR and the TORS ground station 
(coherency/non-coherency, doppler compensation 
on/off, power mode, actual TORS in use, antennae 
configurations); discrepancies in telemetry rate or 
format; inactive or non-locked links; and degrading 
or critical automatic gain control situations (signal 
strength). 

The rule -based method of knowledge representation 
has proven to be quite powerful for this application. 
Rules provide a direct method of encoding the 
fault-isolation knowledge of a spacecraft analyst. 

The development of CLEAR would have taken 
much longer using conventional, non-rule-based 
programming techniques. Perhaps more 
importantly, the rule -based method of representation 
has provided the flexibility to easily adapt the 
knowledge base to unforeseen changes in the 
operational behavior of the spacecraft. For example, 
even though the operational nature of COBE was 
fairly accurately understood by the design engineers 
and flight operations team before the launch, slight 
behavioral variations and complications arose once 
the spacecraft was in orbit. Although the FOAs 
were able to adjust to such variations rather quickly, 
ground monitoring software systems required 
complex modifications. However, the required 
changes to CLEAR’s rule-base were relatively 
straightforward and quickly implemented. After this 
modification, CLEAR provided consistent 
operational assistance. This situation demonstrates 
one of the advantages of the separation of 
knowledge and data in rule-based expert systems. 

Although CLEAR has demonstrated the utility and 
potential of an expert system in the demanding 
environment of a satellite control center, it has also 
revealed many of the pitfalls and deficiencies of the 
development of expert systems. One of the lessons 
learned from this and other initial expert system 
projects is that prototypes can often be developed 
quite rapidly, but operational expert systems require 
considerable effort. 

Early in CLEAR’s development, the primary 
concern was the perceived difficulty of the 
knowledge acquisition effort. However, the 
knowledge engineering task was found to be 
relatively straightforward, albeit time-consuming. 
The development of the rule base was a lengthy 
process due to the interactive nature of the 
knowledge acquisition. Basically, the expert would 
describe a specific piece of knowledge to the 
“knowledge engineer” who would transcribe it into a 


rule, pass it back to the expert for validation, test it, 
and then, finally, release it for operational use. The 
involvement of various players in this process 
'resulted in long turnaround times from the point at 
which a piece of knowledge was determined to be 
important until it was translated into a rule and 
placed into operation. Later in the project, it was 
determined that the translation of this type of 
expertise into rules is quite straightforward and can 
be easily performed by the expert himself. 

The CLEAR development team learned that most of 
the development time for the system was spent on 
issues not directly related to the construction of the 
expert system and its rulebase. A surprising amount 
of the effort focused on the integration of the expert 
system with the data source and graphics display 
system. This required in-depth programming 
knowledge of the interfacing systems and the ability 
to trouble shoot problems within them. Tools are 
needed to simplify the complicated task of 
integrating expert systems with interfacing systems. 

CLEAR is regarded as a successful attempt to 
automate a control center function using an expert 
system; several other missions have requested 
systems similar to it. Although this system is 
beneficial to the COBE flight operations analysts, 
additional benefits can be captured through 
retrospective analysis of the development process 
and focused application to future systems. The 
project described below represents the first steps 
taken that capitalize on a number of the lessons 
learned from the development of the CLEAR Expert 
System. 


The GenSAA Approach 

Partly due to the success of CLEAR, a considerable 
number of expert systems will be developed to 
support control center operations in upcoming 
missions during the early 1990’s. To facilitate the 
development of these systems, a project has been 
initiated to develop an integrated, domain-specific 
tool, named GenSAA (Generic Spacecraft Analyst 
Assistant), that will allow spacecraft analysts to 
rapidly create simple expert systems without having 
to directly deal with the complicated details of the 
systems with which the expert system would have to 
interface. In addition, this tool will allow the expert 
system developer to utilize and/or modify previously 
developed rule bases and system components, thus 
facilitating software reuse and reducing 
development time and effort. 
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Figure 1- Elements of GenSAA 


The GenSAA tool will consist of a development 
environment and system components (figure 1). 

The system components comprise of: 

- the inference engine, 

- the display driver, and 

- a process that manages the reception of data. 

The development environment is composed of three 
utilities: 

- Data Interface Development Utility, 

- Rule Base Development Utility, and 

- User Interface Development Utility. 

Collectively, these utilities will be used to create or 
modify an instance of an expert system. 

The GenSAA development utilities will utilize a 
highly graphical, point-and-select method of 
interaction to facilitate use. The expert system 
developer will use the data interface development 
utility to select the telemetry parameters to be 
monitored, the rule base development utility to 
define the rules which will act on the values of these 
telemetry parameters, and the user interface 
development utility to layout a simple graphical 
representation of the subsystem or process being 
monitored and where the results of the rule 
executions will be displayed. 


The components generated by the development 


utilities are called application-specific components. 
They will be integrated with the GenSAA System 
Components to create a GenSAA Application. A 
GenSAA Application is an expert system that will 
be executed during spacecraft contacts to monitor 
the selected telemetry parameters and to notify the 
flight operations analysts of faults inferred from this 
data. 

To demonstrate the advantage of software reuse and 
to involve the user in the tool definition, the project 
team decided to initially focus on a class of missions 
managed by Goddard. A study of upcoming 
missions was conducted to identify a series of 
missions that have sufficient commonality to enable 
reuse of expert system software from mission to 
mission. The Small Explorer (SMEX) family of 
missions was determined to be an ideal target group 
due to the appropriate time frame of this program, 
the low-cost nature of the missions, the emphasis on 
system reuse, and the rapid turnaround between 
missions. All of these factors correlate closely with 
GenSAA’s objectives. 

GenSAA is intended to be used by FOAs in a 
POCC. In a n effort to monitor the health and safety 
of a satellite and its instruments, FOAs monitor real 
time data looking for combinations of telemetry 
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parameter values, trends, and other indications that 
may signify a problem or failure. The expert 
systems created with GenSAA will greatly assist the 
flight operations analysts with the tedious task of 
data monitoring thereby allowing them to focus on 
other, higher-level responsibilities during the 
real-time contacts with the satellite. This, in turn, 
will likely result in a more efficient and effective 
system of operations. 

The behavior of a satellite is quite dynamic and 
often not well understood until the spacecraft is 
placed in orbit. To quickly create expert systems 
that can effectively monitor satellites, tools are 
needed that allow the analysts to formulate the 
rulebase easily without the intervention or delay of 
knowledge engineers and programmers. By 
eliminating these traditional developers, several 
benefits are expected. The analysts will be able to 
create rules quickly in response to unforeseen 
changes in spacecraft behavior or operational 
procedures. Also, knowledge translation errors will 
be reduced or, at least, more easily corrected. 
Knowledge translation errors are errors which are 
inadvertently introduced during the process of 
translating a piece of expert knowledge into rule 
form. 

In addition to assisting the FOAs with real-time 
spacecraft operations, GenSAA will be useful as a 
training tool in two ways. First, by utilizing the 
playback utilities provided by the new control center 
ground system named TPOCC (Transportable 
Payload Operations Control Center), analysts will be 
able to replay a previous spacecraft communications 
event. Thus, a student analyst can observe how the 
expert system handles a specific problem scenario. 
Exercises like this will provide a realistic, hands-on 
environment for training flight operations analysts in 
a safe, off-line mode. Second, the development of 
rules used in an expert system is a beneficial mental 
training exercise for the FOA. Experience from 
previous expert system projects indicate that the 
actual formation of rules is a beneficial exercise in 
itself. By allowing the analysts to create rules 
themselves, they are forced to consider the 
alternatives more closely thus promoting a deeper 
understanding of the problem domain. This may 
allow the optimal method of fault isolation to be 
identified. 

Another benefit of automating fault-isolation tasks 
with rule -based systems is that the resulting rulebase 
serves as accurate documentation of the 
fault-isolation method. Not only can the rulebase be 


studied by student analysts to learn about 
fault-isolation techniques, but, more importantly, 
mission operations can be better protected against 
the effects of personnel turnovers. POCC expert 
systems that capture fault-isolation knowledge 
preserve expertise from mission to mission and 
mitigate the impact of the loss of experienced, flight 
operations analysts. 

Conclusion 

As satellites become more complex, their operation 
is becoming increasingly difficult. Flight operations 
analysts who are responsible for the command, 
control, health and safety of these spacecraft are 
rapidly being inundated with the data coming at 
them at higher and higher rates. Understandably, 
they are quickly reaching a level of information 
saturation. 

As demonstrated by the CLEAR Expert System, 
fault-isolation expert systems can help flight 
operations analysts monitor the flood of data. These 
systems can accurately monitor hundreds of 
real-time telemetry parameters, isolating 
discrepancies and anomalies the instant they can be 
detected, and alerting the analysts while providing 
advice on how to correct the problems swiftly and 
effectively. However, although these expert systems 
can be quite beneficial, the development of these 
systems is usually time consuming and costly, and 
the resulting system often cannot be easily reused by 
another mission. 

Consequently, GenSAA is being developed for use 
by the flight operations analysts who work in 
satellite control centers. It is designed to provide 
quick and easy development of fault-isolation expert 
systems without the delay or costs of knowledge 
engineers and programmers. By facilitating the 
reuse of expert system elements from mission to 
mission, GenSAA will reduce development costs, 
preserve expertise between missions and during 
periods of personnel turnover, and provide a more 
accurate degree of command and control of our 
rapidly advancing satellites. 
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